A data processing system uses a business application connected to a data network to receive and process data from a number of endpoint data sources. The business application processes the data in accordance with specific business functions. Examples of business applications for such data processing systems include service automation systems and telephone service provisioning systems.
Business applications in data processing systems must often process very large amounts of data, which is made more difficult in applications in which the data cannot be stopped. In a service automation system, an enterprise, such as a computer manufacturer, may collect data from its customers to keep track of the quality of its products, maintenance needs, service requirements, hardware and software configurations, patch history, and any other item of information that the enterprise may wish to track. The data, also known as “telemetry,” is typically text-based and formatted, and depending on the complexity of the specific data, it may be quite voluminous. Computer manufacturers may also have so many products in the field that their service automation systems could be receiving telemetry constantly. Similarly, a telephone provisioning system may receive requests for services, requests for ringtones, requests for pictures, data describing a user, data describing a handset, or data in many forms at virtually any time.
When the data is received, the data typically goes through a parsing phase to convert the text-based nature of the data into an operational data model of the application. Each data processing system may employ its own parsing and data model. Each system may also implement input data models that define the structure of the text data and output data models, which define the operational format of the data as required by the business application.
Traditionally, the input data models and output data models are defined at compile-time. This causes problems that are based on the very nature of the data typically processed by data processing systems. The data of interest evolves in content and in format. The format of the data changes as new information is sought by or presented to the business applications. Many data processing systems must necessarily deal with multiple data formats as it is. For example, a computer manufacturer makes different models of computers based on the different operating systems they run. Each model may present data to the service automation system in a different format.
In addition, input data models and output data models tend to grow in size. As the models grow in size, the differences between the input data models and the output data models grow exponentially requiring increasingly sophisticated and complex parsers.
The increasing complexity of data processing systems is aggravated as increasing load leads to a lack of scalability in the system. This is typically addressed by fragmenting the business application into separate solutions that map to divisions with the overall data model. In many applications, such as service automation systems, most of the information has significant associations. Maintenance of these associations is essential to be able to derive new information about the customer's environment. In addition, the typical solution may cause significant duplication and ultimately siloing of technologies.
In view of the above, a need has long existed for a method and a system that overcome the problems noted above and others previously experienced.